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(54) Apparatus, method and computer program product for decoding and reproducing moving 
images, time control method and multimedia information receiving apparatus 



(57) In image processor section (24), moving image 
signal decoder section (246) decodes the moving image 
bit stream and the time stamp to be used for reproduc- 
tion transmitted from multiple separator section (22) on 
a frame by frame basis and stored in input buffer (245) 
and provides time control section (247) with the time 
stamp to be used for reproduction that corresponds to 
the decoded frame and the information on the time 
stamp in the header of the moving image bit stream, 



while storing the decoded image data in frame memory 
(248). The time control section (247) transmits a request 
to image output section (249) for outputting image data 
at the time specified by the time stamp to be used for 
reproduction as transmitted from the moving image sig- 
nal decoder section (246) or at the time as determined 
on the basis of the time stamp of the moving image itself 
and causes the image data to be read out of the frame 
memory (248) and displayed on the LCD (34) by way of 
LCD control section (25). 
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Description 

[0001 ] This invention relates to a multimedia informa- 
tion receiving apparatus for receiving multimedia infor- 
mation prepared by multiplexing an audio signal and a 5 
moving image signal (moving picture signal or video sig- 
nal) that are related to each other and compressed sep- 
arately, separating and expanding the audio signal and 
the moving image signal and reproducing them synchro- 
nously. More particularly, the present invention relates 
to an improvement to a moving image decoding and re- 
producing apparatus adapted to be used as multimedia 
information receiving apparatus, a moving image de- 
coding and reproducing method, a time control method 
and a computer program product for decoding and re- 
producing a moving image. 

[0002] Generally, in a multimedia information trans- 
mission system adapted to transmit multimedia informa- 
tion obtained by multiplexing an audio signal and a mov- 
ing image signal, the audio encoder and the moving im- 
age encoder of the transmitter are required to process 
respectively the sound and the moving image in a syn- 
chronized manner whilethe audio decoder and the mov- 
ing image decoder of the receiver are required to proc- 
ess the respective output signals so that the sound and 
the moving image may be reproduced in a synchronized 
manner. In order to make it possible to reproduce the 
sound and the moving image synchronously, the inter- 
national coding standards such as MPEG-1 and MPEG- 
2 (MPEG: Moving Pictures Experts Group) provide the 
use of presentation time stamps (PTSs) as information 
for controlling the timings of signal outputs so that both 
the audio signal and the moving image signal are repro- 
duced and output when they are synchronized in the re- 
spective decoders at a given time as controlled by a sys- 
tem time clock (STC). 

[0003] For example, when an audio signal and a mov- 
ing image signal are multiplexed according to MPEG-2, 
the bit stream obtained by encoding the audio signal and 
the moving image signal is divided into groups, which 
are then packetized to produce packets referred to as 
PES (packetized elementary stream) packets having a 
variable length. At this time, a PTS is added to each of 
the PES packets. If a PES packet contains a moving 
image bit stream of a plurality of frames, only a PTS cor- 
responding to the first frame is added to the PES packet. 
In other words, none of the remaining frames are pro- 
vided with a PTS added thereto. 
[0004] However, with the moving image encoding 
method of MPEG-2, there scarcely exists a moving im- 
age bit stream of a plurality of frames in a PES packet 
because the moving image bit stream of a frame is very 
long. Additionally, since any two consecutive frames are 
separated from each other by a constant interval, it is 
possible to accurately estimate the display time of each 
of the remaining frames from the PTS of the first frame 
if each and every frame is not provided with PTS. 
[0005] Meanwhile, the operation of stipulating the 



MPEG-4 Standards for encoding moving images for the 
purpose of mobile communications (radio communica- 
tions) at a low transmission rate is underway. The com- 
ing MPEG-4 video coding system will provide the use of 
a time information added to the header of each VOP 
(video object plane) frame in a bit stream forthe purpose 
of indicating the time for reproducing the VOP. Note, 
however, the time information is produced not by using 
the system time clock but by using a clock whose accu- 
racy is specific to video. 

[0006] Now, let us assume that the moving image bit 
stream of a frame is packetized into a single PES pack- 
et. If the moving image bit stream of a frame is short, 
the PES packet will also be short. However, the over- 
head (the additional data added for the purpose of mul- 
tiplexing) will become large relative to the packet length 
because of the addition of a PTS of real data of 33 bits 
corresponding to the frame in the PES packet. Then, the 
net result will be a lowered overall transmission efficien- 
cy. 

[0007] Then, it may be conceivable to packetize a 
moving image bit stream of a plurality of frames into a 
PES packet and add only a PTS corresponding to the 
first frame in the PES packet. However, according to 
MPEG-4, two consecutive frames are not necessarily 
separated by a constant interval. Then , it will not be pos- 
sible to accurately estimate the display time of each of 
the remaining frames because only the first frame in the 
PES packet is provided with PTS. 
[0008] As pointed out above, with MPEG-4 for mobile 
communications at a low transmission rate, if a moving 
image bit stream is P ES packetized on a frame by frame 
basis and a PTS is added to the frame of each packet, 
the overhead will become too large relative to the packet 
length. Additionally, if a moving image bit steam of a plu- 
rality of frames is put into a single packet and only a PTS 
added to the heading frame in the packet, it will no longer 
be possible to accurately estimate the display time of 
each of the remaining frames. 

[0009] In view of the above identified circumstances, 
it is therefore the project of the present invention to pro- 
vide a moving image decoding and reproducing appa- 
ratus, a moving image decoding and reproducing meth- 
od, a time control method, a computer program product 
configured to decode/reproduce a moving image and a 
multimedia information receiving apparatus that are 
adapted to reproduce both the moving image and the 
sound in an accurately synchronized manner even in a 
situation where a frame has a variable length and a mov- 
ing image coding system of adding the time control in- 
formation of the moving image itself is employed to the 
moving image bit stream of al! the frames, while the 
moving image bit stream is packetized for each frame 
and then the moving image bit streams of a plurality of 
frames are collectively packetized. 
[0010] In an aspect of the invention, there is provided 
a moving image decoding and reproducing apparatus 
configured to decode and reproduce moving image 
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packets, the apparatus comprising: a separator config- 
ured to receive moving image packets sequentially and 
separate each of the moving image packets into a first 
segment or first and second segments of a bit stream 
and a time stamp, the first bit stream segment relating 
to the stamp and having a first image signal correspond- 
ing to one image frame, and the second bit stream seg- 
ment having a second frame image signal correspond- 
ing to another one image frame and time information re- 
lating to the frame image signal; an input buffer config- 
ured to store temporarily the first and second bit stream 
segments and the corresponding time stamps; a decod- 
er configured to read out the first and second bit stream 
segments sequentially and decode the first and second 
bit stream segments into the first and second frame im- 
age signals, respectively; a memory configured to store 
the first and second frame image signals supplied from 
the decoder; an output section configured to output the 
frame image signals from the memory sequentially; and 
a controller configured to calculate reproduction timings 
of outputting the frame image signals from the time 
stamp and the time information, and control the output 
of the frame image signals from the output section in 
accordance with the reproduction timings. 
[0011] In another aspect of the invention, there is pro- 
vided a method of decoding and reproducing moving im- 
age packets, the method comprising: receiving the mov- 
ing image packets sequentially and separating each of 
the moving image packets into a first segment or first 
and second segments of a bit stream and a time stamp, 
the first bit stream segment relating to the time stamp 
and having a first image signal corresponding to one im- 
age frame, and the second bit stream segment having 
a second frame image signal corresponding to another 
one image frame and having time information relating 
to the second frame image signal; storing the first and 
second bit stream segments and the corresponding time 
stamp, temporarily; reading out the first and second bit 
stream segments sequentially and decoding the first 
and second bit stream segments into the first and sec- 
ond frame image signals, respectively; storing the first 
and second frame image signals; outputting the first and 
second stored frame image signals sequentially; and 
calculating, a first reproduction timing of outputting the 
first frame image signal from the time stamp, and a sec- 
ond reproduction timing of outputting the second frame 
image signal from the time stamp and the time informa- 
tion, and controlling the output of the first and second 
frame image signals from the output section in accord- 
ance with the first and second reproduction timings. 
[0012] In still another aspect of the invention, there is 
provided a method of controlling the reproduction timing 
of moving image packets, the moving image packet in- 
cluding first and second segments of a bit stream and a 
time stamp, the first bit stream segment relating to the 
time stamp and having a second frame image signal cor- 
responding to another one image frame and having time 
information relating to the second frame image signal; 



the method comprising: determining a first reproduction 
timing of outputting the first frame image signal from the 
time stamp, and a second reproduction timing of output- 
ting the second frame image signal from the time stamp 
5 and the time information, and controlling the output of 
the first and second frame image signals from the output 
section in accordance with the first and second repro- 
duction timings. 

[0013] In still another aspect of the invention, there is 

10 provided a program product for processing moving im- 
age packets, the moving image packet including first 
and second segments of a bit stream and a time stamp, 
the first bit stream segment relating to the time stamp 
and having afirst image signal corresponding to one im- 

*5 age frame, and the second bit stream segment having 
a second frame image signal corresponding to another 
one image frame and having time information relating 
to the second frame image signal; the program product 
comprising: a program code for determining a first re- 

20 production timing of outputting the first frame image sig- 
nal from the time stamp, and a second reproduction tim- 
ing of outputting the second frame image signal from the 
time stamp and the time information, and controlling the 
output of the first and second frame image signals, re- 

25 spectively; storing the first and second frame image sig- 
nals; outputting the first and second stored frame image 
signals sequentially; and calculating a first reproduction 
timing of outputting the first frame image signal from the 
time stamp, and a second timing of outputting the sec- 

30 ond frame image signal from the time stamp and the 
time information, and controlling the output of the first 
and second frame image signals from the output section 
in accordance with the first and second reproduction 
timings. 

35 [0014] In a further aspect of the invention, there is pro- 
vided a multimedia information receiving apparatus to 
be used in a system configured to transmit the multime- 
dia information formed by packetizing and multiplexing 
the moving image bit stream and the sound bit stream 

40 obtained by encoding a moving image signal and a 
sound signal related to each other on a frame by frame 
basis; each moving image packet of the multimedia in- 
formation being formed by adding time information to be 
used for reproducing to the frame header of the moving 

45 image bit stream, packetizing the moving image bit 
stream for every one or more than one frames and add- 
ing time stamp to be used for reproduction to each pack- 
et of the bit stream of one or more than one frames in 
order to synchronize the moving image signal and the 

so sound signal for reproduction; the apparatus compris- 
ing; a separator configured to separate the moving im- 
age bit stream, the sound bit stream and the time stamps 
to be used for reproduction corresponding respectively 
to the moving image bit stream and the sou nd bit stream ; 

55 a sound decoder and reproducer configured to decode 
the sound bit stream as separated by the separator, and 
reproduce and output the decoded sound signal on the 
basis of the time stamp to be used for sound reproduc- 
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tion; and a moving image decoder and reproducer con- 
figured to decode the moving image bit stream as sep- 
arated by the separator, to receive the time stamp to be 
used for reproduction as separated by the separator and 
to output, for the moving image bit stream of each frame 
having the time stamp added thereto, the decoded mov- 
ing image signal at the time of reproduction as specified 
by the time stamp and, for the moving image bit stream 
of each frame having no time stamp as added thereto, 
the decoded moving image signal at the time of repro- 
duction as determined on the basis of the time stamp as 
detected prior to that frame and the time information to 
be used for reproducing as added to the header of the 
moving image bit stream of the frame. 
[0015] By providing such a moving image decoding 
and reproducing apparatus, a moving image decoding 
and reproducing method, a time control method and a 
computer program product for decoding and reproduc- 
ing a moving image, it is now possible to reproduce both 
the moving image and the sound in an accurately syn- 
chronized manner even in a situation where the moving 
image bit stream of a frame has a variable length and a 
moving image coding system of adding the time control 
information to be mainly used for reproducing of the 
moving image itself is employed to the moving image bit 
stream of all the frames, while the moving image bit 
stream is packetized for each frame and then the mov- 
ing image bit streams of are plurality of frames a collec- 
tively packetized. 

[0016] Additionally by providing such a multimedia in- 
formation receiving apparatus, it is now possible to re- 
produce both the moving image and the sound in an ac- 
curately synchronized manner even in a situation where 
the packets of given multimedia information contains a 
moving image bit stream and the moving image bit 
stream of a frame has a variable length, a moving image 
coding system of adding the time control information to 
be mainly used for reproducing of the moving image it- 
self being employed to the moving image bit stream of 
all the frames, while the moving image bit stream is 
packetized for each frame and then the moving image 
bit streams of a plurality of frames are collectively pack- 
etized. 

[0017] The time of reproduction is controlled by the 
reference time of the system. Then, the signal to be re- 
produced can be synchronized with another signal to be 
reproduced. 

[0018] When time control information to be used for 
reproduction corresponding to a plurality of frames is 
added to the packet, the decoded moving image signal 
of the frame to which the time control signal to be used 
for reproduction is added will be selectively output. 
Then, the frame to which time control information to be 
used for reproduction is added can be randomly ac- 
cessed. 

[0019] In other words, when a moving image coding 
system conforming to MPEG-4 is used, the fact that a 
time stamp for reproducing is added to the header of the 



moving image bit stream will be utilized. Thus, if a time 
stamp to be used for reproduction is added to a system 
packet on a frame-by-frame basis, the time stamp is 
used as reproduction time information. On the other 
5 hand, if a time stamp to be used for reproduction is not 
added to a system packet, the time stamp to be used 
for reproducing as added to the frame header of the 
moving image bit stream is used as reproduction time 
information. 

w [0020] This summary of the invention does not nec- 
essarily describe all necessary features so that the in- 
vention may also be a sub-combination of these de- 
scribed features. 

[0021] The invention can be more fully understood 
15 from the following detailed description when taken in 
conjunction with the accompanying drawings, in which: 

FIG. 1 is a schematic block diagram of multimedia 
information receiving apparatus according to the 
20 embodiment, illustrating the entire configuration 
thereof, as applied to a mobile communication ter- 
minal; 

FIG. 2 is a schematic block diagram of the image 
processor section of the embodiment of FIG. 1 ; 

25 FIGS. 3A, 3B and 3C are schematic illustrations of 
three different moving image bit streams that can 
be used for the purpose of the embodiment, illus- 
trating three different methods to be used by the 
multiplexer/separator section of FIG. 1 for adding a 

30 time stamp to be used for reproduction to a moving 
image bit stream when generating a PES packet; 
FIG. 4 is a flow chart of the processing operation of 
the time control section of FIG. 1 for determining the 
time of displaying the image represented by a mov- 

35 jng image signal; 

FIG. 5 is a timing chart for outputting image data, 
illustrating the first embodiment of moving image re- 
producing operation; 

FIG. 6 is a timing chart for outputting image data, 
40 illustrating the second embodiment of moving im- 
age reproducing operation; and 
FIG. 7 is a timing chart for outputting image data, 
illustrating the third embodiment of moving image 
reproducing operation. 

45 

[0022] Now, the present invention will be described by 
referring to the accompanying drawing that illustrates 
preferred embodiments of the invention. 
[0023] FIG. 1 is a schematic block diagram of an em- 

50 bodiment of multimedia information receiving apparatus 
according to the invention, illustrating the entire config- 
uration thereof, as applied to a mobile communication 
terminal such as cellulartelephone. The mobile commu- 
nication terminal comprises a radio section 1 , a base 

55 band section 2, an input/output section 3 and a power 
source section 4. The multimedia information transmit- 
ted by this system has a format conforming to the 
MPEG-4 Standard. 
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[0024] Referring to FIG. 1 , the radio frequency signal 
getting to the mobile communication terminal from a 
base station (not shown) by way of a radio line of the 
mobile communication system is received by antenna 
11 and then fed to reception circuit (RX) 13 by way of 5 
an antenna duplexer (DUP) 12. The reception circuit 1 3 
comprises a high frequency amplifier, a frequency con- 
verter and a demodulator. The received radio frequency 
signal is subjected to low noise amplification by a low 
noise amplifier and then mixed with a reception local os- 
cillation signal generated by frequency synthesizer 
(SYN) 1 4 in thef requency converter for a frequency con- 
version that turns the radio frequency signal into a re- 
ception intermediate frequency signal or a reception 
base band signal. The output signal is then digitally de- 
modulated by the demodulator. An orthogonal demod- 
ulation method corresponding to the QPSK method may 
typically be used for the demodulation. The frequency 
of the reception local oscillation signal generated by the 
frequency synthesizer 14 is specified by the main con- 
trol section 21 provided in the base band section 2. 
[0025] The demodulated signal output from the recep- 
tion circuit 13 is then input to the base band section 2. 
The base band section 2 comprises a main control sec- 
tion 21 , a multiple separator section 22, a speech coder/ 
decoder (to be referred to as codec hereinafter) 23, an 
image processor section 24, an LCD (liquid crystal dis- 
play) control section 25 and an STC (system time clock) 
generator section 26. The STC is the reference clock in 
the receiver side and may typically be a clock with a fre- 
quency of 27 MHz and synchronized with the counter- 
part of the transmitter side by phase regulation. 
[0026] The main control section 21 identifies the de- 
modulated signal fed from the reception circuit 1 3 to find 
out if it is control information or multimedia information. 
If it is found to be multimedia information, the main con- 
trol section 21 forwards the signal to the multiple sepa- 
rator section 22, which separates the input multimedia 
information into a sound data packet and an image data 
packet. 

[0027] The sound data packet is then supplied to the 
speech codec 23. Upon decoding the sound signal in 
the input sound data packet, the speech codec 23 trans- 
mits the sound signal at a timing that is determined by 
the STC fed from the STC generator section 26 and the 
PTS added to the sound data packet. The sound signal 
output from the speech codec 23 is fed to a loudspeaker 
32 of the input/output section 3 and the original sound 
is acoustically reproduced from the loudspeaker 32. 
[0028] On the other hand, the image data packet is 
supplied to the image processor section 24. Upon de- 
coding the image signal in the input image data packet, 
the image processor section 24 transmits the image sig- 
nal at a timing that is determined by the STC fed from 
the STC generator section 26 and the PTS added to the 
image data packet. The image signal output from the 
image processor section 24 is fed to a LCD 34 of the 
input/output section 3 by way of the LCD control section 



25 and the original image is displayed on the display 
screen of the LCD 34. 

[0029] The LCD control section 25 has the function of 
displaying various pieces of information on the opera- 
tion of the apparatus as output from the main control 
section 21 including a telephone directory, the detected 
intensity of the received electric field and the remaining 
charge on the battery on the LCD 34. 
[0030] The speech signal of the user output from the 
microphone 31 of the input/output section 3 is input to 
the speech codec 23 of the base band section 2 and 
coded there before fed to the multiple separator section 
22. On the other hand, the image signal output from the 
camera (CAM) 33 is supplied to the image processor 
section 24 of the base band section 2 and coded there 
according to MPEG-4 before fed to the multiple separa- 
tor section 22. The multiple separator section 22 multi- 
plexes the coded speech data and the coded image data 
in a predetermined format defined in the MPEG-2 sys- 
tem and generates a base band transmission data. The 
multiplexed transmission data is the fed to the transmis- 
sion circuit (TX) 15 of the radio section 1 from the main 
control section 21 . 

[0031] The transmission circuit 15 comprises a mod- 
ulator, a frequency converter and a transmission power 
amplifier. The transmission data is digitally modulated 
by the modulator and mixed with the transmission local 
oscillation signal generated by the frequency synthesiz- 
er 1 4 in the frequency converter for a frequency conver- 
sion that turns the data into a radio frequency signal. 
The QPSK method may typically be used for the mod- 
ulation. The generated transmission radio frequency 
signal is then amplified to a predetermined transmission 
level and supplied to the antenna 1 1 by way of the an- 
tenna duplexer 12, which antenna 11 transmits the sig- 
nal to the base station (not shown). 
[0032] The power source section 4 comprises a bat- 
tery 41 such as a lithium ion battery, a charging circuit 
42 for electrically charging the battery 41 and a voltage 
generation circuit (PS) 43. The voltage generation cir- 
cuit 43 typically comprises a DC/DC converter and gen- 
erates a predetermined supply voltage Vcc according to 
the output voltage of the battery 41 . 
[0033] The input/output section 4 is provided with a 
lighting unit 36 for lighting the LCD 34 and the key input 
section 35 at the time of user operation and communi- 
cation. The lighting unit 36 is also referred to as backlight 
or illumination. 

[0034] The main control section 21 comprises a mi- 
crocomputer and an internal memory that may include 
a ROM and a RAM. It has an ordinary control feature of 
controlling the connection of radio channels established 
by using appropriate ones of the above components and 
also controlling the communication that takes place after 
the establishment of a communication link. 
[0035] FIG. 2 is a schematic block diagram of the im- 
age processor section 24 of the embodiment of FIG. 1 . 
[0036] Referring to FIG. 2, the moving image signal 
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output from the camera 33 is digitized by the image input 
section 241 and then stored in the frame memory 242. 
The moving image signal encoder section 243 com- 
presses and encodes the moving image data stored in 
the frame memory 242 by means of the moving image 5 
coding method of MPEG-4 and generates a moving im- 
age bit stream, adding a time information to each frame 
as header information. The moving image bit stream 
output from the moving image encoder section 243 is 
then fed to the multiple separator section 22 by way of 
the output buffer 244 and multiplexed with the speech 
bit stream of the speech that is input by way of the mi- 
crophone and encoded by the speech codec 23. Then, 
the multiplexed signals are transmitted as multimedia 
information. 

[0037] The multiple separator section 22 is adapted 
to packetize the signal input to it on a frame-by-frame 
basis or by a plurality of frames to generate PES pack- 
ets. At this time, a time stamp (PTS) to be used for re- 
production that is synchronized with the sound is added 
to each PES packet. PES packets may be formed from 
the moving image bit stream on a VOP (frame) by VOP 
(frame) basis or by a plurality of VOP frames. 
[0038] The multiple separator section 22 converts the 
PES packets into TS packets having a fixed length. A 
PCR (program clock reference) is added to the head of 
eachTS packet in order to synchronize the system clock 
of the transmitter side and that of the receiver side. 
[0039] As the multimedia information that is generat- 
ed in a manner as described above is received, the mul- 
tiple separator section 22 restores the PES packets from 
the received TS packets and separates the sound bit 
stream and the moving image bit stream from each of 
the PES packets. Then, it transmits the sound bit stream 
to the speech codec 23 to reproduce the original sound 
from the loudspeaker 32 while it sends out the moving 
image bit stream to the image processor section 24. At 
this time, the time stamp (PTS) that has been added to 
the PES packet is then added to the moving image bit 
stream. 

[0040] A time stamp (PTS) may be added to the head 
of each and every piece of VOP header information 
(VOP stamp code) as shown in FIG. 3A when the mov- 
ing image bit stream is packetized into PES packets on 
a frame-by-frame basis. Alternatively, a time stamp 
(PTS) may be added only to the head of the leading 
piece of VOP header information as shown in FIG. 3B 
when the moving image bit stream of n frames is collec- 
tively packetized into a PES packet. Still alternatively, a 
time stamp (PTS) may be added to the head of every 
intermediary piece of VOP header information as shown 
in FIG. 3C. It should be noted that, in FIGS. 3A, 3B and 
3C, a VOP data refers to the moving image bit stream 
of a frame. 

[0041 ] The multiple separator section 22 forwards the 
PCR of the TS packet it receives to the STC generator 
section 26. The STC generator section 26 corrects its 
own clock so as to be synchronized with that of the trans- 



mitter side on the basis of the PCR it receives and sends 
it to the speech codec 23 and the image processor sec- 
tion 24 so that it may be referred to for synchronized 
reproduction. 

[0042] The image processor section 24 stores the 
moving image bit stream and the time stamp (PTS) it 
receives from the multiple separator section 22 in the 
input buffer245. Then, the moving image signal decoder 
section 246 decodes the moving image bit stream and 
the time stamp (PTS) stored in the input buffer 245 and 
provides the time control section 247 with the time 
stamp (PTS) and the time information indicating the time 
of reproduction as contained in the header information 
of the moving image bit stream corresponding to the de- 
coded frame, while it stores the decoded image data in 
the frame memory 248. 

[0043] The STC corrected by the SCR provided by the 
multiple separator section 22 is then given to the time 
control section 247. The time control section 247 trans- 
mits a request for providing an image data to the image 
output section 249 at the time when the STC comes to 
agree with the time stamp (PTS) for reproduction from 
the moving image signal decoder section 246 or when 
the time corresponding to the one indicated by the time 
information of the moving image itself has elapsed. 
[0044] Upon receiving the request for outputting an 
image data from the time control section 247, the image 
output section 249 reads out the image data from the 
frame memory 248 and causes the LCD 34 to display 
the image data by way of the LCD control section 25. 
Then, it transmits a notification of completion of the out- 
put to the time control section 247. Upon receiving the 
notification of completion of the output from the image 
output section 249, the time control section 247 trans- 
mits a reading request to the input buffer 245 in order to 
cause the moving image signal decoder section 246 to 
start decoding the next frame. 

[0045] FIG. 4 is a flow chart of the processing opera- 
tion of the time control section 247 for controlling the 
time of displaying a frame. Note that, in the below de- 
scription, B-TS is a time information that indicates the 
time of reproduction as contained in the VOP header 
information of the moving image bit stream. In addition, 
TB (cur) is the time determined on the basis of the B-TS 
corresponding to the current VOP. On the other hand, 
TS (cur) is a time information that indicates the time de- 
termined on the basis of the time stamp (PTS) that indi- 
cates the time of reproduction acquired from a PES 
packet and corresponds to the current VOP. Although 
the B-TS always exists corresponding to each of the 
VOP, the PTS does not always exists corresponding to 
each of the VOP. Therefore, the time notion "TS (prev) 
and TB (TS (prev))" are introduced. The TS (prev) is the 
time determined on the basis of the previous PTS and 
the TB (TS (prev)) is the time determined on the basis 
of B-TS of VOP corresponding to the previous PTS. 
[0046] Referring to FIG. 4, firstly, it determines the 
time TB (Cur) to be referred to for the current frame on 



15 



20 



25 



30 



35 



40 



45 



50 



6 



11 



EP 1 195 996 A2 



12 



the basis of the time information to be used for repro- 
ducing B-TS that is contained in the VOP header infor- 
mation (Step S1 ). Then, it determines if a time stamp to 
be used for reproduction (PTS) is added to the head of 
the VOP header information or not (Step S2). 
[0047] If it is determined in Step S2 that a time stamp 
to be used for reproduction (PTS) is added (Yes), the 
time control section 247 determines the time of repro- 
duction of the current frame TS (Cur) from the time 
stamp to be used for reproduction (PTS) (Step S3) and 
updates the time of reproduction of the preceding frame 
(Prev) and the reference time TB (TS (Prev)) (TS (Prev) 
= TS (Cur), TB (TS (Prev) = TB (Cur)) (Step S4). Sub- 
sequently, the time control section 247 sets the display 
time to T = TS (Cur) and terminates the sequence of 
time control operation. If, on the other hand, it is deter- 
mined in Step S2 that a time stamp to be used for re- 
production (PTS) is not added (No), thetimecontrol sec- 
tion 247 sets the display time to T - TS (Prev) + TB (Cur) 
- TB (TS (Prev)) and terminates the sequence of time 
control operation. 

[0048] The moving image reproducing operation of 
the image processor section 24 having the above de- 
scribed configuration will now be described by way of 
the first through third embodiments. 
[0049] As the first embodiment, assume that PES 
packets are formed on a frame by frame basis and a 
time stamp (PTS) is added to the head of each and every 
piece of VOP header information (VOP start code) as 
shown in FIG. 3A. FIG. 5 is a timing chart for outputting 
image data, illustrating the first embodiment of moving 
image reproducing operation. 

[0050] Also assume thatthe time stamps added to the 
PES packets are PTS #0, PTS #2, PTS #1 , «. and the 
times of reproduction provided by them are respectively 
TS0, TS1 , — With this arrangement, all the frames are 
provided with respective time stamps (PTS) so that the 
image of the frame #0 is displayed when the time t as 
indicated by the system clock (STC) is t ^ TS0 (= TO), 
the image of the frame #1 is displayed when the time t 
as indicated by the system clock (STC) is t ^ TS1 (= 
T1), and so on. 

[0051] As the second embodiment, assume a PES 
packet is formed for n frames and a time stamp (PTS) 
is added only to the head of first piece of VOP header 
information as shown in FIG. 3B. FIG. 6 is a timing chart 
for outputting image data, illustrating the second em- 
bodiment of moving image reproducing operation. 
[0052] Also assume that the time of the time stamp to 
be used for reproduction PTS #0 as added to the VOP 
header information of the leading frame #0 is TS0, that 
of the time information B-TS #0 contained in the VOP 
header information of the frame #0 is TB0, that of the 
time information B-TS #1 contained in the VOP header 
information of the frame #1 is TB1 and so on. 
[0053] With this arrangement, firstly, as the time 
stamp PTS #1 for determining the time of reproduction 
TS0 of the frame #0 is provided, the image of the frame 



#1 is displayed when the reference time t becomes t ^ 
TS0 (= TO). Then, since the time stamp PTS for deter- 
mining the time of reproduction of the frame #1 is not 
provided, the time information B-TS #1 to be used for 
5 reproduction as contained in the VOP header informa- 
tion of the frame #1 is utilized. Thus, the time of display 
T1 of the frame #1 is determined by equation 

10 T1 = TS0 + (TB1 - TB0) 

and the image of the frame #1 is displayed at the time 
when t ^ T1 holds true. A similar procedure is followed 
subsequently. If the time of the time information B-TS 
15 #n to be used for reproduction as contained in the VOP 
header information of the frame #n is TBn, the time Tn 
(n being an integer satisfying n ^ 1 ) at which the image 
of the frame #n is displayed is determined by equation 

20 

Tn = TS0 + (TBn - TB0) 

and the image of the frame #n is displayed at the time 
when t ^ Tn holds true. 
25 [0054] In the case of this embodiment, since it is safe 
to assume that TS0 is the initial delay time for displaying 
the moving image, it is possible to accurately define the 
initial delay time in a system for synchronously multi- 
plexing an audio signal and a moving image signal. 
30 [0055] Finally, as the third embodiment, assume a 
PES packet is formed for n frames and a time stamp 
(PTS) is added not only to the head of first piece of VOP 
header information but also to the head of intermediary 
pieces of VOP header information as shown in FIG. 3C. 
35 Note that a time stamp (PTS) may be added to the head 
of intermediary pieces of VOP header information at 
regular time intervals such as every 2 seconds or to the 
head of the piece of VOP header information of I picture 
(I - VOP). According to MPEG-4, a moving image bit 
40 stream comprises a part referred to as configuration in- 
formation (CI) and a part referred to as elementary 
stream (ES) and one or more than one pieces of CI are 
provided for every ES, although a piece of CI may be 
inserted to a moving image bit stream many times. 
45 Therefore, a time stamp (PTS) may alternatively be add- 
ed to the head of a piece of VOP header information 
located immediately after an inserted piece of CI. 
[0056] FIG. 7 is a timing chart for outputting image da- 
ta, illustrating the third embodiment of moving image re- 
50 producing operation. Also assume here that, like the 
second embodiment, the time stamp (PTS0) is added 
to the frame #0 and the time information (TSm) is added 
to the head of the piece of VOP header information of 
the frame #m (m being an integer satisfying 1 < m < n), 
55 the following equations hold true; 

Tm-1 = TS0 + (TBm-1 - TB0), 
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Tm = TSm 

and 

Tn = TSm + (TBn - TBm), 

where Tm-1 , Tm andTn respectively represent the time 
at which the image of the frame #m-1 is displayed, the 
time at which the image of the frame #m is displayed 
and the time at which the image of the frame #n is dis- 
played. Then, in the frame #m-1 , the display time is com- 
puted by referring to the time TSO of the time stamp to 
be used for reproduction PTS #0 of the frame #0 and 
the time TBm-1 of the time information B-TS #m-1 to be 
used reproducing as contained in the piece of VOP 
header information of the frame #m-1 and the image is 
displayed when t ^ Tm-1 holds true. In the frame #m, 
the time TSm of the time stamp to be used for reproduc- 
tion PTS #m is used so that the image of the frame #m 
is displayed when t ^ Tm holds true. Then, in the frame 
#n, the display time is computed by referring to the time 
stamp to be used for reproduction PTS m of the frame 
#m and the time TBn indicated by the time information 
in the piece of VOP header information of the frame #n 
and the image of the frame #n is displayed when t ^ Tn 
holds true. 

[0057] In the case of this embodiment, time stamp er- 
rors can occur in the moving image bit stream and im- 
ages can be displayed according to wrong pieces of time 
information particularly if signals are transmitted by way 
of a low quality transmission path such as that of a mo- 
bile communication system (radio communication sys- 
tem). However, wrong pieces of time information are re- 
placed by highly reliable pieces of time information of 
the system as the time stamps to be used for reproduc- 
tion (PTS) that are added to intermediary parts of a PES 
packet are used on a priority basis. 
[0058] Additionally, when a presentation time stamp 
to be used for reproduction (PTS) is inserted between 
two frames in such a manner that time stamps (PTS) 
are added for frames in a packet, respectively, as in the 
case of this embodiment, it may be so arranged that the 
time stamps to be used for reproduction (PTS) are re- 
trieved and one of the retrieved time stamps that is add- 
ed for the frame of the moving image signal is selectively 
used to start outputting the reproduced signal. With this 
arrangement, the frame to which the time stamp to be 
used for reproduction (PTS) is added can be randomly 
accessed. 

[0059] In this random access operation, a selective 
output mode can be set on the key input section 35. 
When the key input section 35 is depressed to set the 
selective output mode, a selective mode signal is gen- 
erated in the key input section 35 and is supplied to the 
main control section 21 to set the main control section 
21 . Thus, the main control section 21 sets the time con- 



trol section 247 in the selective output mode and the 
time control section 247 controls the output section 249. 
That is, the time control section 247 determines selec- 
tive output timings of the frame image signals in the out- 
put section 249 in accordance with the time stamps so 
that the frame image signals that are provided with the 
time stamps (PTS) as described above are selectively 
output from the output section 249 in accordance with 
the output timings. 

[0060] Thus, the image processor section 24 of the 
above embodiment determines if a time stamp to be 
used for reproduction (PTS) is added to the moving im- 
age bit stream on a frame by frame basis or not in the 
process of decoding and reproducing the image signal 
and, if it is found that time stamps to be used for repro- 
duction (PTS) are added to the moving image bit stream 
on a frame by frame basis, it uses not the time informa- 
tion added to the header information of the moving im- 
age bit stream but the time stamps to be used for repro- 
duction (PTS) as reproduction time information. On the 
other hand, if it is found that time stamps to be used for 
reproduction (PTS) are not added to the moving image 
bit stream on a frame by frame basis, it obtain reproduc- 
tion time information for the frames, using the time in- 
formation added to the header information of the moving 
image bit stream. Therefore, it is now no longer neces- 
sary to assign a time stamp to each and every frame so 
that the overhead of each frame of a moving image sig- 
nal can be reduced. Additionally, the present invention 
provide an effect of making it possible to randomly ac- 
cess the frame that is adapted to establish the initial de- 
lay time and to which a time stamp to be used for repro- 
duction (PTS) is added. 

[0061] While the present invention is applied to the 
image processor section of a mobile communication ter- 
minal in the above embodiment, the present invention 
is by no means limited thereto. In other words, the 
present invention can be applied to a receiving appara- 
tus of a multimedia information transmission system 
conforming to MPEG-4 or similar standards. 



Claims 

1 . A moving image decoding and reproducing appara- 
tus configured to decode and reproduce moving im- 
age packets, the apparatus characterized by com- 
prising: 

a separator (22) configured to receive moving 
image packets sequentially and separate each 
of the moving image packets into a first seg- 
ment or first and second segments of a bit 
stream and a time stamp, the first bit stream 
segment relating to the stamp and having a first 
image signal corresponding to one image 
frame, and the second bit stream segment hav- 
ing a second frame image signal corresponding 
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to another one image frame and time informa- 
tion relating to the frame image signal; 
an input buffer (245) configured to store tempo- 
rarily the first and second bit stream segments 
and the corresponding time stamps; 5 
a decoder (246) configured to read out the first 
and second bit stream segments sequentially 
and decode the first and second bit stream seg- 
ments into the first and second frame image 
signals, respectively; 10 
a memory (248) configured to store the first and 
second frame image signals supplied from the 
decoder (246); 

an output section (249) configured to output the 
frame image signals from the memory (248) se- '5 
quentially; and 

a controller (247) configured to calculate repro- 
duction timings of outputting the frame image 
signals from the time stamp and the time infor- 
mation, and control the output of the frame im- 20 
age signals from the output section (249) in ac- 
cordance with the reproduction timings. 

The apparatus according to claim 1 , characterized 
by further comprising; 25 

a timer (26) configured to generate a reference 
timing signal; 

wherein said controller (247) controls the out- 
put section (249) with reference to the refer- 30 
ence timing signal. 

The apparatus according to claim 1 , characterized 
by further comprising; 

35 

an input section (35) configured to set a prede- 
termined output mode to select outputs of the 
frame image signals; 

wherein said controller (247) is configured to 
set said output section (249) in response to the *o 
predetermined output mode to determine se- 
lective output timings corresponding to the time 
stamp, and control the output of the frame im- 
age signals from the output section (249) in ac- 
cordance with the selective output timings. 45 

A method of decoding and reproducing moving im- 
age packets, said method characterized by com- 
prising: 

50 

receiving the moving image packets sequen- 
tially and separating each of the moving image 
packets into a first segment or first and second 
segments of a bit stream and a time stamp, the 
first bit stream segment relating to the time 55 
stamp and having a first image signal corre- 
sponding to one image frame, and the second 
bit stream segment having a second frame im- 



age signal corresponding to another one image 
frame and having time information relating to 
the second frame image signal; 
storing the first and second bit stream seg- 
ments and the corresponding time stamp, tem- 
porarily; 

reading out the first and second bit stream seg- 
ments sequentially and decoding the first and 
second bit stream segments into the first and 
second frame image signals, respectively; 
storing the first and second frame image sig- 
nals; 

outputting the first and second stored frame im- 
age signals sequentially; and 
calculating, a first reproduction timing of output- 
ting the first frame image signal from the time 
stamp, and a second reproduction timing of out- 
putting the second frame image signal from the 
time stamp and the time information, and con- 
trolling the output of the first and second frame 
image signals from the output section in ac- 
cordance with the first and second reproduction 
timings. 

5. A method of controlling the reproduction timing of 
moving image packets, said moving image packet 
including first and second segments of a bit stream 
and a time stamp, the first bit stream segment re- 
lating to the time stamp and having a second frame 
image signal corresponding to another one image 
frame and having time information relating to the 
second frame image signal; 

said method characterized by comprising: 

determining a time to be referred to for the cur- 
rent frame at the time of reproducing said mov- 
ing image signal from the time control informa- 
tion to be used for reproducing; 
determining a first reproduction timing of out- 
putting the first frame image signal from the 
time stamp, and a second reproduction timing 
of outputting the second frame image signal 
from the time stamp and the time information, 
and controlling the output of the first and sec- 
ond frame image signals from the output sec- 
tion in accordance with the first and second re- 
production timings. 

6. A program product for processing moving image 
packets, said moving image packet including first 
and second segments of a bit stream and a time 
stamp, the first bit stream segment relating to the 
time stamp and having a first image signal corre- 
sponding to one image frame, and the second bit 
stream segment having a second frame image sig- 
nal corresponding to another one image frame and 
having time information relating to the second frame 
image signal; 
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said program product comprising: 

a program code for determining a first repro- 
duction timing of outputting the first frame im- 
age signal from the time stamp, and a second 5 
reproduction timing of outputting the second 
frame image signal from the time stamp and the 
time information, and controlling the output of 
the first and second frame image signals, re- 
spectively; 10 
storing the first and second frame image sig- 
nals; 

outputting the first and second stored frame im- 
age signals sequentially; and 
calculating a first reproduction timing of output- *5 
ting the first frame image signal from the time 
stamp, and a second timing of outputting the 
second frame image signal from the time stamp 
and the time information, and controlling the 
output of the first and second frame image sig- 20 
nals from the output section in accordance with 
the first and second reproduction timings. 

7. A multimedia information receiving apparatus to be 
used in a system configured to transmit the multi- 25 
media information formed by packetizing and mul- 
tiplexing the moving image bit stream and the sound 
bit stream obtained by encoding a moving image 
signal and a sound signal related to each other on 
a frame by frame basis; 30 

each moving image packet of said multimedia 
information being formed by adding time infor- 
mation to be used for reproducing to the frame 
header of said moving image bit stream, pack- 35 
etizing the moving image bit stream for every 
one or more than one frames and adding time 
stamp to be used for reproduction to each pack- 
et of the bit stream of one or more than one 
frames in order to synchronize the moving im- 40 
age signal and the sound signal for reproduc- 
tion; said apparatus comprising; 
a separator (22) configured to separate the 
moving image bit stream, the sound bit stream 
and the time stamps to be used for reproduction 45 
corresponding respectively to the moving im- 
age bit stream and the sound bit stream; 
a sound decoder and reproducer (23) config- 
ured to decode the sound bit stream as sepa- 
rated by said separator (22), and reproduce so 
and output the decoded sound signal on the ba- 
sis of the time stamp to be used for sound re- 
production; and 

a moving image decoder and reproducer (24) 
configured to decode the moving image bit 55 
stream as separated by said separator (22) ; to 
receive the time stamp to be used for reproduc- 
tion as separated by said separator (22) and to 



output, for the moving image bit stream of each 
frame having the time stamp added thereto, the 
decoded moving image signal at the time of re- 
production as specified by said time stamp and, 
for the moving image bit stream of each frame 
having no time stamp as added thereto, the de- 
coded moving image signal at the time of repro- 
duction as determined on the basis of the time 
stamp as detected prior to that frame and the 
time information to be used for reproducing as 
added to said header of the moving image bit 
stream of the frame. 

8. The apparatus according to claim 7, characterized 
In that 

said moving image decoder and reproducer 
(24) includes : 

an input buffer (245) configured to store tempo- 
rarily the moving image bit stream and the time 
stamp to be used for reproduction as separated 
by said separator (22); 

a moving image signal decoder (246) config- 
ured to read out the moving image bit stream 
from the input buffer (245) on a frame by frame 
basis and decode the moving image bit stream 
at the time specified by the time information to 
be used for reproducing as added to the frame 
header of each frame; 

a frame memory (248) configured to store mov- 
ing image signals corresponding to each frame 
decoded by said moving image signal decoder 
(246); 

an image output section (249) configured to 
output the moving image signal stored in the 
frame memory (248) according to the instruc- 
tion for reproduction; and 
a time controller (247) configured to receive the 
time information to be used for reproducing as 
obtained by said moving image signal decoder 
(246) and the stamp to be used for reproduction 
as obtained by said separator (22) as input and 
direct, with respect to the frame having time 
stamp as added thereto, said moving image 
output section (249) to output the correspond- 
ing moving image signal at the reproduction 
time as specified by the information and, with 
respect to the frame having no time stamp as 
added thereto, said moving image output sec- 
tion (249) to output the corresponding moving 
image signal at the reproduction time as deter- 
mined from the time stamp added to the pre- 
ceding frames and the time information of the 
frame. 

9. The apparatus according to claim 8, characterized 
in that 
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said time controller (247) controls the reproduc- 
tion time on the basis of the reference time of 
the system transmitting the multimedia infor- 
mation. 

5 

10. The apparatus according to claim 8, 
characterized in that 

when the time stamp to be used for reproduc- 
tion is added to the moving image bit stream of a 
plurality of frames in said moving image packet, 10 
said image output section (249) selectively outputs 
the decoded moving image signal of the frame hav- 
ing the time stamp. 
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